Method for calibrating vehicular vision system

ABSTRACT

A method for calibrating a vehicular vision system includes providing a camera at a vehicle, with the camera having a field of view. Images are captured with the camera and a set of resultant images are acquired for a classification. Information is extracted related to image features in the set of resultant images, and an appropriate subset of coefficients is determined. For each classification, a classification vector of at least one appropriate weight is stored that corresponds to the determined subset of coefficients. The determined subset of coefficients is determined by processing sets of coefficients produced from a selection of calibration images and determining a subset of coefficients which acceptably discriminate between defined classifications. A set of resultant images is acquired by limiting the dynamic range of acquired images to obtain resultant images that include at least one region of interest.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 14/076,524, filed Nov. 11, 2013, now U.S. Pat. No. 9,077,962,which is a continuation of U.S. patent application Ser. No. 12/529,832,filed Sep. 3, 2009, now U.S. Pat. No. 8,581,983, which is a 371 nationalphase application of PCT Application No. PCT/CA2008/000477, filed Mar.7, 2008, which claims the priority benefit of U.S. provisionalapplication Ser. No. 60/893,477, filed Mar. 7, 2007.

FIELD OF THE INVENTION

The present invention relates to a system and method for determininginformation relating to the interior of a vehicle and/or its contents.More specifically, the present invention relates to a system and methodfor determining a classification relating to the interior of thevehicle.

BACKGROUND OF THE INVENTION

Many passenger and other vehicles are now equipped with supplementalrestraint systems (SRS), such as front or side airbags, to protectvehicle occupants in the event of an accident. However, while such SRScan in many cases prevent or mitigate the harm which would otherwiseoccur to a vehicle occupant in an accident situation, in somecircumstances it is contemplated that they can exacerbate the injury toa vehicle occupant. Specifically, SRS such as airbags must deployrapidly, in the event of an accident, and this rapid deploymentgenerates a significant amount of force that can be transferred to theoccupant. In particular, children and smaller adults can be injured bythe deployment of airbags as they both weigh less than full sized adultsand/or they may contact a deploying airbag with different parts of theirbodies than would a larger adult.

For these reasons, regulatory agencies have specified the operation anddeployment of SRS. More recently, regulatory bodies, such as theNational Highway Transportation and Safety Administration (NHTSA) in theUnited States, have mandated that vehicles be equipped with a devicethat can automatically inhibit deployment of the passenger airbag incertain circumstances, such as the presence of a child in the passengerseat or the seat being empty.

To date, such devices have been implemented in a variety of manners, themost common being a gel-filled pouch in the seat base with an attachedpressure sensor which determines the weight of a person in the passengerseat and, based upon that measured weight, either inhibits or permitsthe deployment of the airbag. However, such systems are subject toseveral problems including the inability to distinguish between anobject placed on the seat and people on the seat, the presence of childbooster/restraint seats, etc.

It has been proposed that image-based sensor systems could solve many ofthe problems of identifying and/or classifying occupants of a vehicle tocontrol SRS but, to date, no such system has been developed which canreliably make such determinations in real world circumstances whereinlighting conditions, the range of object variability, materials andsurface coverings and environmental factors can seriously impede theability of the previously proposed image-based systems from making areliable classification.

It has also previously been proposed that image-based systems andmethods may be useful in classifying matters such as a measure of driveralertness, by acquiring and processing images of the driver within theinterior of the vehicle, or classifying the presence of passengerswithin the vehicle allowing for the optimized control of vehicleenvironmental systems (such as air conditioning) and/or entertainmentsystems by classifying the occupancy of one or more vehicle seats.However, to date, it has proven difficult to achieve a desired level ofreliability for such systems.

It is desired to have an image-based system and method that candetermine a classification relating to the interior of the vehicle, suchas the occupancy status of a vehicle seat, from one or more images ofthe vehicle interior.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a novel system andmethod of determining a classification relating to the interior of thevehicle which obviates or mitigates at least one disadvantage of theprior art.

According to a first aspect of the present invention, there is provideda method for determining a classification relating to the interior of avehicle, comprising the steps of: (i) with an image capture device,acquiring a resultant image of a portion of the vehicle interior whichis of interest; (ii) extracting information relating to a set of imagefeatures from the acquired resultant image; (iii) statisticallyprocessing the extracted information with a previously determined set ofimage feature information values, each member of the set of imagefeature information values corresponding to a respective one of a set ofpredefined classifications relating to the interior of the vehicle, todetermine the most probable classification; and (iv) outputting thedetermined most probable classification related to the interior of thevehicle.

Preferably, step (ii) comprises processing the resultant image with atwo dimensional complex discrete wavelet transform to produce a selectedset of coefficients related to features in the acquired resultant imageand, in step (iii), the previously determined set of image featurevalues comprises a set of weights for each defined classification, theweights being multiplied with the set of coefficients to produce a scorefor each defined classification, the defined classification with thehighest produced score being the classification output in step (iv).

Also preferably, the determined classification relates to the occupancyof a vehicle seat and the portion of the vehicle interior which is ofinterest includes the portion of the vehicle seat which would beoccupied by a passenger in the vehicle.

According to another aspect of the present invention, there is provideda system for determining a classification relating to the interior of avehicle, comprising: an image capture device operable to acquire animage of a portion of the vehicle interior which is of interest; animage capture subsystem operable to process the image to limit thedynamic range of the image to obtain a resultant image; an image featureextractor operable to produce for the resultant image a set of valuescorresponding to features in the resultant image; and a classifieroperable to combine the set of values produced by the image featureextractor with a predetermined set of classification values, eachclassification value corresponding to a different possible predefinedclassification, the results of this combination representing theprobability that each predefined classification is the currentclassification, the classifier operable to select and output the mostprobable classification.

The present invention also provides a vehicle interior classificationsystem and method which determines a classification relating to theinterior of the vehicle, such as the occupancy status of a vehicle seator the state of alertness of a vehicle driver, from one or more imagesof an appropriate portion of the interior of the vehicle acquired withan image capture device.

The acquired images are preferably processed to limit the dynamic rangeof the images to obtain a resultant image which can comprise one or moreregions of interest which are less than the total field of view of theimage capture device.

The resultant images are processed to extract information about featuresin the image and, in one embodiment, this processing is achieved with atwo-dimensional complex discrete wavelet transform which produces a setof coefficients corresponding to the presence and/or location of thefeatures in the resultant image.

The set of coefficients produced with such a transform is potentiallyquite large and can be reduced, through described techniques, to asubset of the total number of coefficients, the members of the subsetbeing selected for their ability to discriminate between theclassifications defined for the system.

By selecting a subset of the possible coefficients, computationalrequirements are reduced, as are hardware requirements in the system,such as memory.

The selected set of coefficients (whether comprising all of thecoefficients or a subset thereof) are provided to a classifier whichprocesses the coefficients with a set of calibration vectors, that weredetermined when the system was calibrated, to determine the mostprobable classification for the portion of the vehicle interior.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the present invention will now be described, byway of example only, with reference to the attached Figures, wherein:

FIG. 1 is a block diagram representation of a classification system inaccordance with the present invention;

FIG. 2 is a flowchart showing a method of calibrating the system of FIG.1; and

FIG. 3 is a flowchart showing a method of operating the system of FIG.1.

DETAILED DESCRIPTION OF THE INVENTION

A vehicle interior classification system in accordance with the presentinvention is indicated generally at 20 in FIG. 1. System 20 includes animage capture device 24 which can be any suitable device or system forcapturing an image, or sequence of images, of the portion of interest ofthe interior of the vehicle in which system 20 is installed. In thefollowing discussion, a specific implementation of the present inventionis described, wherein a determination of the occupancy status of avehicle seat is obtained.

However, as will be apparent to those of skill in the art, the presentinvention is not so limited and can be employed to determine a widerange of classifications, based upon images of at least portions of theinterior of a vehicle. For example, a classification of driver alertnesscan be determined with other embodiments of the present invention bycapturing images of the driver seat occupant and surrounding area.

Examples of suitable image capture devices 24 include CMOS and/or CCDcamera systems with at least one image sensor and an appropriate set ofoptical lenses and/or filters so that a suitable image of a portion ofinterest in the interior of the vehicle can be obtained. In a presentembodiment of the invention, image capture device 24 acquires grayscaleimages but it is contemplated that color images can also be employed insome circumstances if desired. In the present embodiment, image capturedevice is a CMOS monochrome MT9V022 system, manufactured by MicronTechnology, Inc., 8000 S. Federal Way, Boise, Id., USA and this systemproduces an image with a resolution of one-hundred and eighty-eightpixels by one-hundred and twenty pixels.

It is also contemplated that image capture device 24 can be a time offlight (To F) imaging device. Such devices are known and use thedifference in the polarization of the source light and the reflectedlight from imaged objects to determine image depth information. It iscontemplated that, if desired, image capture device 24 can acquireimages using ToF techniques or, more preferably, that image capturedevice 24 can acquire both a ToF-derived image and a conventionallyacquired image. In such a case, the image (or a portion of it) acquiredby ToF can be employed in addition to the image acquired with theconventional imaging techniques.

Image capture device 24 is located in the vehicle interior at a positionwhereby the portion of the interior which is of interest can be imaged.For the specific example of determining a classification the occupant aseat, the occupant seating portion of the vehicle seat underconsideration will be imaged. In many circumstances, it will be desiredto classify the occupancy of the front passenger seat, but it is alsocontemplated that it may be desired to classify the occupancy of rearseats of a vehicle to control side air bag SRS or other systems.

To date, in order to capture images of the occupant seating portion ofpassenger seats, the present inventors have successfully located imagecapture device 24 in the roof liner, the A pillar and/or in the overheadconsole of different vehicles. However, as will be apparent to those ofskill in the art, the present invention is not limited to image capturedevice 24 being located in any of the these three positions and,instead, image capture device 24 can be located in any suitable positionas will occur to those of skill in the art provided that the selectedposition allows for the acquisition of images of the portion of interestin the vehicle interior. For example, for classifying driver alertness,image capture device 24 can be located in the A pillar adjacent thedriver, the roof liner or the dashboard instrument cluster, etc. One ofthe challenges of using image-based technologies in a vehicle is thewide range of lighting conditions that the system must cope with.

Lighting conditions ranging from direct sunlight, to overcast sunlightto nighttime conditions must all be accommodated by image capture device24. Further, the dynamic range of the captured images can be very largeas part of the image may be in direct sunlight while another part may bein shade. To deal with images with a high dynamic range, many availableCCD and/or CMOS camera systems provide high dynamic range (HDR)functions which process captured images by taking multiple images atdifferent imaging sensitivities and combing appropriate portions ofthese images to acquire a single resultant image CIResultant) with areduced dynamic range. Imaging devices and/or systems with such HDRfunctions are well known and are commercially available, and CCD or CMOScamera systems with HDR functions can be employed as image capturedevice 24 in the present invention. In such a case, the HDR processed!Resultant image is employed by system 20, as described further below.

However, in a presently preferred embodiment of the invention, system 20does not employ an HDR function but instead employs an image subtractionprocess to acquire acceptable resultant images with image capture device24. Specifically, image capture device 24 is connected to an imagecapture subsystem 28 as is an auxiliary light source 32. Auxiliary lightsource 32 comprises one or more sources of Near Infrared (NIR) light(i.e.—light wavelengths in the range from about 700 nanometers to about1500 nanometers), such as NIR emitting LEDs.

Auxiliary light source 32 is positioned within the vehicle such that theemitted light will illuminate the region of interest of the vehicleinterior. Subsystem 28 controls auxiliary light source 32 and canactivate or deactivate auxiliary light source 32 as needed. Imagecapture device 24 includes a filter system which allows image capturedevice 24 to capture images using visible light and the NIR lightemitted by auxiliary light source 32, while preferably blocking otherundesired light such as far infrared light and/or UV light which mayotherwise degrade the acquired images. To capture an image for use insystem 20 in the presently preferred embodiment of the invention, imagecapture subsystem 28 activates image capture device 24 to acquire oneimage (!Ambient) of the region of interest of the vehicle interiorilluminated with the ambient light in the vehicle. The captured Ambientimage is stored in a memory in image capture subsystem 28.

Next, image capture subsystem 28 activates auxiliary light source 32 toilluminate the occupant seating portion of the vehicle seat with NIRlight, in addition to the ambient light, and activates image capturedevice 24 to acquire a second image 0Ambient+NIR) of the region ofinterest of the vehicle interior.

The first, IAmbient image is then subtracted from the second,IAmbient+NIR image by image capture subsystem 28. Provided that !Ambientand IAmbient+NIR were acquired with little time passage between theimage capture operations, large changes in the ambient light conditionsbetween capture of IAmbient and IAmbient+NIR are avoided and thisresults in an image which is effectively an image acquired with NIRlight only (INIR) and which can be employed as the resultant imageIResultant in system 20.

!Resultant=INIR=IAmbient+NIR−!Ambient

In a present embodiment, image capture and processing speeds of five ormore resultant images per second have been easily achieved. Thus theinfluences and effects of ambient lighting within the vehicle aremitigated and images with large dynamic ranges are avoided.

The use of NIR light in auxiliary light source 32 is presently preferredas: NIR light is invisible to the human eye, thus having no effect onthe passengers of the vehicle and ambient sunlight tends to haverelatively little light in the NIR wavelengths as these wavelengths arereadily absorbed by moisture in the atmosphere. However, as will beapparent to those of skill in the art, the present invention is notlimited to the use of light in NIR wavelengths for auxiliary lightsource 32 and other light frequencies can be employed for imagesubtraction operations in the present invention, if desired. Further,while the use of image subtraction is presently preferred over HORprocessing of images, the present invention is not limited to the use ofimage subtraction processing to acquire resultant images and HORprocessing or any other suitable manner of obtaining useful resultantimages can be employed as will occur to those of skill in the art.

In addition to controlling image capture device 24, auxiliary lightsource 32 and performing either HOR functions or image subtractionoperations to obtain !Resultant image capture subsystem 28 preferablyalso performs an Occlusion Detection function. Specifically, as system20 requires an image of the region of interest of the interior of thevehicle, any significant occlusion of the field of view of image capturedevice 24 can inhibit proper operation of system 20.

Such significant occlusions can occur, for example, when an occupant ofthe vehicle places a hand, arm or other body portion in a position whichmay occlude a substantial portion of the field of view of image capturedevice 24, or when cargo or luggage is similarly placed, etc.

Accordingly, image capture subsystem 28 preferably includes an OcclusionDetection function to detect unacceptable occlusions in the field ofview of image capture device 24. The Occlusion Detection function can beimplemented in a variety of manners, as will occur to those of skill inthe art, and is not be discussed herein in further detail.

In the event that the Occlusion Detection function detects anunacceptable occlusion in the field of view of image capture device 24,system 20 can provide an alarm or other signal to the vehicle occupantsto indicate the unacceptable occlusion and/or can provide a controlsignal 34 such that system 20 will output a predefined default safeclassification to other appropriate vehicle systems, such as SRS.

Once image capture subsystem 28 has a suitable resultant image, thatresultant image is provided to an image feature extractor 36. Imagefeature extractor 36 operates on the acquired resultant image to produceinformation relating to the occurrence, amplitudes, locations and/orother information relating to features and/or aspects of the resultantimage. In a presently preferred embodiment of the invention, imagefeature extractor 36 performs a two dimensional Discrete WaveletTransform (DWT) on the resultant image to produce a set of coefficientvalues relating to the features within the resultant image. Morespecifically, in this embodiment, image feature extractor 36 employs atwo dimensional Complex Discrete Wavelet Transform (CDWT) to produce thedesired set of coefficient values. DWT's and CDWT's are well known tothose of skill in the art and CDWT's are discussed, for example, in“Image Processing With Complex Wavelets”, Phil. Trans. R. Soc. Land. A,357, 2543-2560, September 1999, by Nick G. Kingsbury, the contents ofwhich are incorporated herein by reference.

In a present embodiment, the resultant image is decomposed by imagefeature extractor 36 using Daubechies filters with the two dimensionalCDWT.

However, it will be understood that any other suitable wavelet filtercan be employed, as will occur to those of skill in the art, such asHaar filters for example. In a present embodiment, the resultant imageis processed to three levels with the two dimensional CDWT as it hasbeen found that three levels of feature extraction provide a reasonableand useful set of information about features in the resultant image.However, the present invention is not limited to three levels ofdecomposition of the resultant image and it is contemplated that evenone level of decomposition can provide worthwhile results in somecircumstances. However, generally, two or more levels of processing arepreferred.

As will be apparent to those of skill in the art, the processing of theacquired resultant image with a two dimensional CDWT produces a largeset of coefficients. In the above-mentioned example of image capturedevice 24 having a resolution of one-hundred and eighty-eight byone-hundred and twenty pixels, for a total of twenty-two thousand,five-hundred and six pixels, a first level of decomposition of theacquired resultant image with a two dimensional CDWT results in fourcoefficients for each pixel, or ninety-thousand, two-hundred and fortycoefficients. A second level of decomposition results in anothertwenty-two thousand, five-hundred and sixty coefficients. A third levelof decomposition results in yet another five-thousand, six-hundred andforty coefficients.

Thus, for the example wherein image capture device has a resolution ofone-hundred and eighty-eight by one-hundred and twenty pixels, threelevels of decomposition of the resultant image with the two dimensionalCDWT results in a total of one-hundred and eighteen thousand,four-hundred and forty coefficients.

As should be apparent, in many circumstances it is desirable to reducethis number of coefficients to a more reasonable number, to reducesubsequent computational requirements and hardware expenses and toobtain a more manageable system. Accordingly, as is described below inmore detail, a representative subset of these coefficients is preferablyselected as elements in a one dimensional array, referred to herein asthe feature vector 40, which represents a selection of the informationrelating to features of interest in the resultant image.

While the present invention can operate on the entire resultant image,the present inventors have determined that advantages can be obtainedwhen the resultant image from image capture subsystem 28 is subdividedinto one or more regions of interest (ROI) and image feature extractor36 only operates on these ROIs. Thus while a ROI can be the entire fieldof view of image capture device 24, it is preferred that one or moresmaller ROIs be defined. The advantages of employing ROIs representingless than the entire field of view of image capture device 24 includereduced computational complexity, as only the portions of the resultantimage which can contribute to a meaningful result are processed by imagefeature extractor 36, and portions of the field of view of image capturedevice 24 which could contain “distractions” (such as, for example,reflections in a side window) that could lead to incorrect results, arenot processed by image feature extractor 36.

For example, if system 20 is determining a classification of theoccupancy of the front passenger seat in a vehicle, one or more ROIs canbe defined which otherwise exclude those portions of the rear vehicleseat and surrounding area and/or portions of the side window area thatare included in the total field of view of image capture device 24.

Accordingly, as part of the setup and calibration of systems 20(described below in more detail) a set of one or more ROIs is definedfor the portion of the vehicle interior which is of interest.

When ROIs are employed, these ROI definitions are used by image capturesubsystem 28 to extract and process only the portions of the capturedimages within the defined ROIs to produce a resultant image containingonly the ROIs. This resultant image is then provided to image featureextractor 36 which produces a correspondingly reduced number ofcoefficients compared to processing the entire field of view (i.e.—fullresolution) of image capture device 24.

For example, in a particular implementation of the present invention fordetermining a classification of the occupant of a seat, the resultantimage is subdivided into three ROIs which comprise about sixty percentof the total pixels of the entire field of view of image capture device24. If the defined three ROIs have a total resolution of aboutthirteen-thousand, five-hundred and thirty-six pixels (i.e.—about sixtypercent of twenty-two thousand, five-hundred and sixty pixels), and theROIs are decomposed to three levels with a two dimensional CDWT, thisresults in about sixty-eight thousand and forty coefficients, ratherthan the one-hundred and eighteen thousand, four-hundred and fortycoefficients which would result from processing the entire field ofview.

Each set of these coefficients, or each of a selected subset of them (asdescribed in more detail below with respect to the setup and calibrationof system 20) comprise the elements of a feature vector 40 whichrepresents the image features of interest to system 20. In a presentembodiment of the invention, image feature extractor 36 is implementedwith a Blackfin™ digital signal processor (DSP), manufactured by AnalogDevices, Inc., Three Technology Way, Norwood, Mass., USA, although anysuitable DSP or other suitable computing device can be employed, as willoccur to those of skill in the art.

Each feature vector 40 from image feature extractor 36 is provided to aclassifier 44 which processes feature vector 40 with a predefinedlibrary 48 of calibration vectors, whose elements comprise weights. Eachcalibration vector in library 48 corresponds to one of a set ofclassifications predefined for system 20 and library 48 is producedduring the setup and calibration of system 20, as described below.

For example, when system 20 is used to classify the occupancy of avehicle seat to control a SRS, a set of possible classifications caninclude “adult”, “child”, “empty seat”, “object”, “child restraintseat”, etc. For driver alertness monitoring systems, possibleclassifications can include “alert”, “impaired”, “attention wandering”,etc.

Classifier 44 combines feature vector 40 with the classification vectorsin library 48, as described below, to obtain a most likelyclassification 52 for the region of interest of the interior of thevehicle imaged by system 20.

Classification 52 is then provided to a decision processor 60, describedin more detail below, which determines an operating classification forthat region of interest and outputs a control signal 64, appropriate forthe determined operating classification, to other vehicle systems.

In the above-mentioned occupancy-based SRS control example, theoperating classification relates to the classification of the occupantof the vehicle seat imaged (or other vehicle interior region beingclassified) by system 20 where, for example, if the operatingclassification corresponds to “child”, signal 64 may be used to inhibitdeployment of the SRS.

As is apparent from the above, the setup and calibration of system 20 isimportant for the correct operation of system 20 and will now bedescribed. As a first step in the setup and calibration of system 20, aset of ROIs are defined for the images captured by image capture device24. As mentioned above, the ROI can comprise the entire field of view ofimage capture device 24, but it is preferred that the field of view ofimage capture device 24 instead be divided into two or more ROIs whichexclude at least part of the field of view of image capture device 24 toreduce processing requirements, reduce hardware expense and, wherepossible, to remove possible sources of “distractions” from theresultant images.

Generally, the ROIs are determined empirically, but in a more or lesscommon-sense manner. For example, if the occupancy status of a vehiclefront passenger seat is to be determined, the region of the capturedimage in which the top of the seat base (on which a passenger rests) andthe front of the seat back (against which a seat occupant's backtouches) are visible, throughout the total permitted range of movementof the seat base and seat back, can be defined as one, or two, ROIs.Similarly, the area adjacent the vertical side of the seat back,throughout its entire permitted range of movement, can be defined asanother ROI.

As will be apparent to those of skill in the art, the present inventionis not limited to the resultant images comprising one, or two, ROIs andthree or more ROIs can be defined as desired.

Once a presumed reasonable set of ROIs has been defined for setup andcalibration, effectively defining the resultant image which will beprocessed, system 20 is operated through a plurality of calibrationscenarios, as described below, each of which scenarios corresponds to aknown one of the classifications to be defined for system 20.

For example, assuming one of the classifications defined for system 20is “adult”, an adult (or an anthropomorphic training dummy (ATD)representing an adult), is placed in the vehicle seat. A resultantimage, comprising the defined ROIs, is obtained of the seat's occupantby image capture device 24 and image capture subsystem 28 and isprocessed by image feature extractor 36. Image feature extractor 36produces a feature vector 40 comprising the set of coefficientsdecomposed from the resultant image and that feature vector isassociated with the appropriate classification, in this case “adult”.

The process is repeated a number of times for the classification beingcalibrated (i.e.—“adult”} with changes being made to the scenario eachtime.

These changes can include changing the adult (or ATD's) position in theseat and/or the seat position, changing the adult (or ATD) for anotheradult-sized person (or ATD) with different dimensions, etc. until arepresentative set of feature vectors has been obtained for the “adult”classification. This may require that one thousand or more calibrationimages be acquired and processed.

The process is then repeated with scenarios appropriate for each otherclassification (e.g.—“empty seat”, “object on seat”, “alert drive”,etc.) to be defined for system 20 to produce an appropriate set offeature vectors for each classification to be defined and calibrated.

As mentioned above, even with the definition of appropriate ROIs andoperating image feature extractor 36 only on pixels within those ROIs,the number of coefficients which results is still very large. To furtherreduce the number of coefficients and to produce the classificationweights for the classification vectors of library 48, the calibrationprocess then employs a statistical regression model to iterativelyidentify the indices of a subset of the total number of coefficients ineach vector resulting from processing the ROIs, where the coefficientsin this subset are sufficient to effectively discriminate between thedefined classifications.

For example, in the example given above, each feature vector comprisessixty-eight thousand and forty coefficients (i.e.—{c1* . . . ,Ces,040}).

The result of the statistical regression process can be theidentification of the indices of a subset of two-thousand, two-hundredof those coefficients, (e.g.—{c213, C503, C2425, . . . , Csg.oos}), andthe production of a set of corresponding regression weights, which caneffectively discriminate between the defined classifications.

In a present embodiment of the invention, the selection of the subset ofcoefficients and production of the regression weights is performed onthe feature vectors obtained from the classification scenarios using aniterative wrapper method, with a “best-first” feature search engine, anda partial least squares regression model as the induction algorithm.

Suitable wrapper methods are well known and one description of aniterative wrapper method is provided in, “The Wrapper Approach” by RonKohavi and George H. John, a chapter in, “Feature Extraction,Construction and Selection: A Data Mining Perspective”, edited by HuanLiu and Hiroshi Motoda and published by Kluwer Academic Press, 1998.Another description is provided in “Wrappers For Feature SubsetSelection (late draft)”, by Ron Kohavi and George H. John, from“Artificial Intelligence Journal, special issue on Relevance”, Vol. 97,Nos. 1-2, pp. 273-324.

The statistical regression model employs a partial least squaresdimension reduction technique, combined with multinomial logisticregression to produce appropriate regression weights for thecoefficients at each selected coefficient index.

As mentioned above, this process is iterative, with different subsets ofcoefficients being considered for their ability to discriminate betweendefined classifications. If the subset of coefficients identified bythis process proves to be less accurate than desired at discriminatingbetween two or more classifications under some conditions, thedefinition of the ROIs can also be changed. In this case, thecalibration scenarios can be re-executed and/or additional calibrationscenarios can be added, the production of feature vectors is performedagain and the statistical regression process is performed again,iteratively, until a satisfactory definition of the ROIs and asatisfactory subset of coefficients, and their corresponding regressionweights, is obtained.

For each defined classification of system 20, a calibration vector ofthe calculated regression weights, corresponding to the indices of theselected subset of coefficients, is stored in library 48 to representeach defined classification of system 20. Thus, if system 20 has sixdefined classifications, library 48 will include six calibrationvectors, each corresponding to one of the defined classifications.Calibration and setup of system 20 is then complete.

Typically, calibration and setup of system 20 must be performed once foreach model of vehicle that system 20 is to be installed in due to thespecific geometries of the interior of the vehicles, the design of theseats employed, etc. Subsequent changes in seat design or other interiorfactors of a model can require recalibration of system 20.

It is contemplated that library 48 can be updated from time to time andreloaded into a vehicle employing system 20, if needed, to allow system20 to handle new classifications, to correct commonly occurringmisclassifications that have been identified subsequent to calibrationand setup, etc.

Once calibration and setup have been completed, image feature extractor36 will only determine values for the subset of coefficients selectedduring calibration. Thus, in the example above, image feature extractor36 will determine values for the about two thousand, two hundredcoefficients in the selected subset and each calibration vector inlibrary 48 includes a like number of regression weights.

As mentioned above, in normal operations (i.e.—once calibration andsetup have been completed) image feature extractor 36 outputs toclassifier 44 a feature vector 40 which comprises the determined valuesfor each of the identified subset of coefficients decomposed from aresultant image.

Classifier 44 receives feature vector 40 and multiplies a copy offeature vector 40 with each of the calibration vectors in library 48 toproduce a corresponding set of scores, one score per definedclassification. Each score indicates the likelihood that the region ofinterest in the vehicle under consideration by system 20 is in arespective one of the defined classifications. The classification whichhas highest determined score is then output, to a decision processor 60,as classification 52.

Decision processor 60 preferably employs a classification (i.e.—state)transition model and a temporal model to filter classifications 52received from classifier 44 to reduce the probability of intermittentmisclassifications.

In particular, transient artifacts in the resultant image and/orpositioning of people and/or objects within the region of interest inthe vehicle under consideration can result in brief errors in theclassification.

Accordingly, decision processor 60 employs classification 52, a historyand a state change table to determine an appropriate outputclassification 64 from system 20.

Specifically, the state change table is a heuristic table that containsvalues which define the probability of transition from each definedstate (i.e. defined classification) of system 20 to each other definedstate. For example, for seat occupancy classifications, the change froman “adult” classification to an “empty seat” classification is morelikely than a direct transition from an “adult” classification to a“child” classification. Accordingly, the state change table contains aset of state change probability values (SCPs) that define theprobability of each state changing to each other state.

Decision processor 60 maintains a numeric confidence level (CL) whichdefines the stability/confidence of the current output classification64.

When system 20 starts, or when decision processor 60 changes outputclassification 64, CL is set to a value of one, which indicates that thecurrent output classification is relatively new, with little priorhistory. Decision processor 60 increments or decrements CL based uponthe values of two counters maintained in decision processor 60.Specifically, decision processor 60 maintains a confidence level up(CLU) counter and a confidence level down (CLO) counter, each of whichhas a predefined minimum value of zero and a predefined maximum value.

When classification 52 from classifier 44 is the same classification asthe current output classification 64, the CLU counter is incremented andthe CLO counter is reset to zero by decision processor 60. Conversely,when classification 52 from classifier 44 is a different classificationthan the current output classification 64, the CLO counter isincremented and the CLU counter is reset to zero by decision processor60.

A confidence level test (CLT) value is predefined against which the CLOand CLU counters are compared. If the CLU counter equals the CLT value,then CL is incremented and the CLU counter is reset to zero. Conversely,if the CLO counter equals the CLT, then CL is decremented and the CLO isreset to zero. If neither the CLO or CLU values are equal to the CLT,the CL remains unchanged. Essentially, the CLT defines the number ofoutput classifications 52 required to occur before a change in theoutput classification 64 can occur.

Finally, a state change value (SCV) is used to effect changes in theoutput classification 64. Specifically, the SCV is the product of the CLand the corresponding SCP in the state change table. For example, if thecurrent output classification 64 is “adult” and the most recentclassification 52 received at decision processor 60 is “empty seat”,then the SCP in the state control table corresponding to a state changefrom “adult” to “empty seat” is multiplied with the current CL to obtaina SCV.

If the most recent classification 52 has been received, unchanged, atdecision processor 60 a number of consecutive times at least equal tothe SCV, then decision processor 60 will change output classification 64to equal that most recent classification 52. Conversely, if the mostrecent classification 52 has been received at decision processor 60 anumber of consecutive times less than the value of the SCV, thendecision processor 60 will not change output classification 64.

As will be apparent from the above, in cases where the CL is high, achange to a new classification for output classification 64 will takelonger than in circumstances wherein the CL is low. Further, byselecting appropriate values for the maximum CL value, the maximumrequired number of consecutive occurrences of a new classification 52 tooccur before a change in the output classification 64 occurs can be setas desired.

It should be understood by those of skill in the art that decisionprocessor 60 can also be responsive to additional inputs, such asocclusion detected signal 34, to alter output classification 64 asnecessary. In the specific case of an occlusion being detected, perhapsfor some minimum amount of time, decision processor 60 can change outputclassification 64 to a predefined default safe classification until theocclusion is removed. A method, in accordance with the presentinvention, is now described, with reference to the flow charts of FIGS.2 and 3. The method of calibrating system 20 is shown in FIG. 2.

The method starts at step 100 where a plurality of resultant images isobtained for each classification to be defined for system 20. Eachresultant image can be an image representing the entire field of view ofthe image capture device employed or can be a composite of one or moredefined Regions of Interest (ROIs) within that field of view. Resultantimages are obtained for a plurality of variations within eachclassification to be defined. For example, the position of an adult orATD on a vehicle seat of interest can be changed and the seat baseand/or seat back positions can be changed throughout a permitted rangeof movements while resultant images are obtained for these variations. Aset of resultant images may comprise as many as one thousand or moreresultant images for each classification.

The acquisition of the resultant images can be performed with highdynamic range processing, or with image subtraction processing, or withany other suitable method of acquiring appropriate images as will occurto those of skill in the art.

At step 104, information related to image features is extracted for eachacquired resultant image. In the presently preferred embodiment of theinvention, image feature extraction is achieved by processing eachresultant image with a two dimensional Complex Discrete WaveletTransform, preferably employing Daubechies filters, to produce a vectorof coefficients representing features of the image for each resultantimage. In the presently preferred embodiment, a three leveldecomposition is employed but it is also contemplated that fewer oradditional levels of decomposition can be performed if desired.

At step 108, an appropriate set of the coefficients of the vectors isselected and a set of weights corresponding to the selected coefficientsis determined. While the selected subset can include all of thecoefficients, it is contemplated that, in most circumstances, it will bedesired to select a subset of less than all of the availablecoefficients.

In a present embodiment of the invention, the selection of theappropriate subset is performed using an iterative wrapper method, witha “best-first” feature search engine, and a partial least squaresregression model as the induction algorithm, although any other suitableselection method and method for determining weights, as will occur tothose of skill in the art, can be employed.

At step 112, for each classification, the set of weights is stored as acalibration vector for the classification, the weights corresponding tothe contribution of each coefficient in the selected subset to thatclassification. If the above-described iterative wrapper method and apartial least squares regression model is employed to determine theselected subset of coefficients, the weights will be the resultingcorresponding regression weights for each defined classification. Ifanother selection method is employed, the weights can be produced in anyappropriate manner as will occur to those of skill in the art. Thecalibration vectors are stored for use by system 20.

The method of operating system 20 is shown in FIG. 3. The method startsat step 200 where an appropriate resultant image is obtained. Theresultant image can be acquired by HOR processing, image subtraction orvia any other suitable method of obtaining an appropriate image, as willoccur to those of skill in the art, provided only that the image shouldbe acquired in the same manner as were the resultant images used in thecalibration of system 20.

The acquired images can comprise the entire field of view of the imagecapture device or can comprise one or more ROIs defined within thatfield of view, again provided only that the ROIs should be the same asthe ROIs employed when acquiring the resultant images used in thecalibration of system 20.

At step 204, image feature extraction is performed on the acquiredresultant image to produce a feature vector representing informationcorresponding to features of interest in the resultant image which areproduced in the same manner as in the calibration operation. In apresent embodiment of the invention, the feature vector comprises thevalues for the subset of coefficients, selected during the calibrationoperation, determined with a two dimensional Complex Discrete WaveletTransform, preferably employing Daubechies filters. Again, othertechniques, as will occur to those of skill in the art, can be used toproduce the feature vector provided only that the feature vector shouldbe obtained in the same manner as were the calibration vectors used inthe calibration of system 20.

At step 208, the feature vector obtained in step 204 is processed withthe set of calibration vectors stored at step 116. In a presentembodiment of the invention, copies of the feature vector are multipliedwith each of the calibration vectors of corresponding weights to producea set of scores, each score representing the probability that theacquired resultant image corresponds to a respective definedclassification. The highest scoring classification is selected as themost probable classification.

At step 212, the most probable classification from step 208 is processedto determine the output classification 64 from system 20. In the presentembodiment, the most probable classification is an input to a decisionprocessor wherein the most probable classification is combined with: thepresent output classification, a confidence level updated and maintainedby the decision processor; and a table of state transition values, thevalues representing the probability of a change from each state(classification) to each other state (classification). The result of thecombination of these values is an output classification which is outputby system 20.

The present invention provides a vehicle interior classification systemand method which determines a classification relating to the interior ofthe vehicle, such as the occupancy status of a vehicle seat or the stateof alertness of a vehicle driver, from one or more images of anappropriate portion of the interior of the vehicle acquired with animage capture device.

The acquired images are preferably processed to limit the dynamic rangeof the images to obtain a resultant image which can comprise one or moreregions of interest which are less than the total field of view of theimage capture device.

The resultant images are processed to extract information about featuresin the image and, in one embodiment, this processing is achieved with atwo-dimensional complex discrete wavelet transform which produces a setof coefficients corresponding to the presence and/or location of thefeatures in the resultant image.

The set of coefficients produced with such a transform is potentiallyquite large and can be reduced, through described techniques, to asubset of the total number of coefficients, the members of the subsetbeing selected for their ability to discriminate between theclassifications defined for the system.

By selecting a subset of the possible coefficients, computationalrequirements are reduced, as are hardware requirements in the system,such as memory.

The selected set of coefficients (whether comprising all of thecoefficients or a subset thereof) are provided to a classifier whichprocesses the coefficients with a set of calibration vectors, that weredetermined when the system was calibrated, to determine the mostprobable classification for the portion of the vehicle interior.

The above-described embodiments of the invention are intended to beexamples of the present invention and alterations and modifications maybe effected thereto, by those of skill in the art, without departingfrom the scope of the invention which is defined solely by the claimsappended hereto.

1. (canceled) 2: A method for calibrating a vehicular vision system,said method comprising: providing a camera at a vehicle, said camerahaving a field of view; capturing images with said camera; acquiring aset of resultant images; extracting information related to imagefeatures in the set of resultant images; acquiring a plurality ofclassifications; determining a subset of coefficients; for eachclassification, storing a classification vector of at least oneappropriate weight that corresponds to the determined subset ofcoefficients; wherein the determined subset of coefficients isdetermined by processing sets of coefficients produced from a selectionof calibration images and determining a subset of coefficients whichacceptably discriminate between defined classifications; and whereinacquiring a set of resultant images comprises limiting the dynamic rangeof acquired images to obtain resultant images that comprise at least oneregion of interest, which encompasses a region of the field of view thatis less than the field of view of said camera. 3: The method of claim 2,wherein said camera has a field of view interior of the vehicle. 4: Themethod of claim 3, wherein the classifications are used to classify anoccupancy of a vehicle seat and include (i) adult, (ii) child, (iii)empty seat, (iv) object and (v) child restraint seat. 5: The method ofclaim 4, wherein, responsive to determining that the extractedinformation corresponds to one of the classifications, a passenger sideairbag control is controlled. 6: The method of claim 2, wherein afeature vector is provided to a classifier that processes the featurevector with a predefined library of calibration vectors corresponding torespective ones of a set of predefined classifications. 7: The method ofclaim 6, wherein the classifier receives the feature vector andmultiplies the feature vector with each of the calibration vectors inthe library of calibration vectors to produce a corresponding score perclassification, and wherein each of the scores indicates the likelihoodthat the region of interest has features within a respective one of thedefined classifications, and wherein the classification having thehighest determined score is then output as the determinedclassification. 8: The method of claim 2, wherein the set of resultantimages comprises images of a region of interest within the field of viewof said camera and wherein the region of interest corresponds to therespective classification for that set of resultant images. 9: Themethod of claim 2, comprising providing occlusion detection to detect anocclusion at least partially occluding the field of view of said camera.10: The method of claim 9, comprising providing an alert responsive todetection of an occlusion. 11: The method of claim 2, wherein theacquired resultant images comprise images of at least two regions ofinterest within the field of view of said camera. 12: The method ofclaim 11, wherein extracting image features in the set of resultantimages is performed on the at least two regions of interest. 13: Themethod of claim 12, wherein images of the set of resultant images areacquired by obtaining a first image illuminated by ambient light and asecond image illuminated by ambient light and a second frequency rangeof light and subtracting the first image from the second image to obtainthe resultant images. 14: A method for calibrating a vehicular visionsystem, said method comprising: providing a camera at a vehicle, saidcamera having a field of view; capturing images with said camera;acquiring a set of resultant images; wherein the acquired resultantimages comprise images of at least two regions of interest within thefield of view of said camera; extracting information related to imagefeatures in the set of resultant images; acquiring a plurality ofclassifications; determining a subset of coefficients; for eachclassification, storing a classification vector of at least oneappropriate weight that corresponds to the determined subset ofcoefficients; providing occlusion detection to detect an unacceptableocclusion at least partially occluding the field of view of said camera;and providing an alert responsive to detection of an occlusion. 15: Themethod of claim 14, wherein said camera has a field of view interior ofthe vehicle, and wherein the classifications are used to classify anoccupancy of a vehicle seat and include (i) adult, (ii) child, (iii)empty seat, (iv) object and (v) child restraint seat. 16: The method ofclaim 15, wherein, responsive to determining that the extractedinformation corresponds to one of the classifications, a passenger sideairbag control is controlled, and wherein, responsive to determiningthat the extracted information is indicative of a child in the seat,actuation of the passenger side airbag is inhibited. 17: The method ofclaim 14, wherein extracting image features in the set of resultantimages is performed on the at least two regions of interest, and whereinimages of the set of resultant images are acquired by obtaining a firstimage illuminated by ambient light and a second image illuminated byambient light and a second frequency range of light and subtracting thefirst image from the second image to obtain the resultant images. 18: Amethod for calibrating a vehicular vision system, said methodcomprising: providing a camera at a vehicle, said camera having a fieldof view; capturing images with said camera; acquiring a set of resultantimages; wherein the set of resultant images comprises images of a regionof interest within the field of view of said camera and wherein theregion of interest corresponds to the respective classification for thatset of resultant images; wherein acquiring a set of resultant imagescomprises limiting the dynamic range of acquired images to obtainresultant images that comprise at least one region of interest, whichencompasses a region of the field of view that is less than the field ofview of said camera; extracting information related to image features inthe set of resultant images; acquiring a plurality of classifications;determining a subset of coefficients; for each classification, storing aclassification vector of at least one appropriate weight thatcorresponds to the determined subset of coefficients; and providing afeature vector to a classifier that processes the feature vector with apredefined library of calibration vectors corresponding to respectiveones of a set of predefined classifications, and wherein the classifierreceives the feature vector and multiplies a copy of feature vector witheach of the calibration vectors in the library of calibration vectors toproduce a corresponding score per classification, and wherein each ofthe scores indicates the likelihood that the region of interest hasfeatures within a respective one of the defined classifications, andwherein the classification having the highest determined score is thenoutput as the determined classification. 19: The method of claim 18,wherein said camera has a field of view interior of the vehicle, andwherein the classifications are used to classify an occupancy of avehicle seat and include (i) adult, (ii) child, (iii) empty seat, (iv)object and (v) child restraint seat, and wherein, responsive todetermining that the extracted information corresponds to one of theclassifications, a passenger side airbag control is controlled, andwherein, responsive to determining that the extracted information isindicative of a child in the seat, actuation of the passenger sideairbag is inhibited. 20: The method of claim 18, wherein images of theset of resultant images are acquired by obtaining a first imageilluminated by ambient light and a second image illuminated by ambientlight and a second frequency range of light and subtracting the firstimage from the second image to obtain the resultant images. 21: Themethod of claim 18, wherein the determined subset of coefficients isdetermined by processing sets of coefficients produced from a selectionof calibration images and determining a subset of coefficients whichacceptably discriminate between defined classifications.